AD~A243 


675 


USACERL  Technical  Report  P-91/52 


October  1991 

T^B:  Test  of  MicroStation  Computer-Based  Instruction 


TECHNOLOGY 
TRANSFER 
TEST  BED 
PROGRAM 


Adaptive  On-Line  Heip 
for  Embedded 
Instructional  Systems 


by 

Doris  Smith  Shaw 
L.  Michael  Golish 
Julie  L.  Webster 
Der-Shung  Yang 


The  U  S.  Army  Corps  of  Engineers  (USAGE)  has  long  recognized 
the  benefits  of  computer-aided  design  (CAD)  systems,  and  has 
encouraged  its  design  architect/engineers  (A/Es)  to  use  this 
technology.  However,  traditional  methods  of  training  A/Es  to  use 
CAD  have  had  only  limited  success,  so  USAGE  has  investigated 
and  developed  various  innovative,  nontraditional  methods.  One 
such  method,  on-line  embedded  instruction,  uses  CAD  software 
as  the  delivery  medium  for  an  on-line  tutorial  program. 

This  report  discusses  an  on-line  tutorial  program  developed  for 
the  Corps’  standard  CAD  system,  Intergraph  MicroStation.  An 
innovative  feature  of  the  Corps’  tutorial  program  is  its  adaptive  on¬ 
line  help  function,  which  includes  both  procedural  and  conceptual 
help  material,  presented  to  the  user  on  the  basis  of  his  or  her 
degree  of  CAD  experience.  The  effectiveness  of  this  tutorial  was 
investigated  in  a  USAGE  Technology  Transfer  Test  Bed  (T^B) 
demonstration,  with  particular  attention  on  the  adaptive  on-line 
help  function. 

Findings  indicate  that  the  adaptive  help  function  was  effective  for 
most  test  participants,  but  it  could  probably  be  modified  to  better 
address  the  needs  of  certain  categories  of  learners. 
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PERFORMING  LABORATORY:  USACERL  PRODUCT/SYSTEM:  MicroStation  Teaching 

Assistant 
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Portland  District 

DESCRIPTION/OBJECTIVE  OF  TEST/DEMONSTRATION: 

USAGE  design  professionals  and  technicians  from  the  three  test  sites  completed  five  CAD  lessons 
on  a  walk-in  basis  using  the  embedded  instruction  program  with  adaptive  on-line  help.  Subjects  were 
observed  for  their  use  of  the  help  function  during  the  lessons  and  were  surveyed  for  their  opinions 
about  the  cffcclivcncss  of  the  method.  The  objective  was  to  investigate  the  effectiveness  of  on-line 
help  in  leaching  a  conceptual  understanding  of  CAD  systems. 

RESULTS  OF  THE  TEST/DEMONSTRATION: 

It  was  concluded  that  the  embedded  instruction  program  was  generally  effective,  but  there  were 
differences  in  the  success  of  certain  subgroups  based  on  their  degree  of  experience  and  whether  or 
not  they  used  help.  Novice  help  users  and  experienced  subjects  who  did  not  use  help  tended  to  feel 
more  successful  and  positive  about  the  tutorial  than  did  the  experienced  help  users  and  novice 
subjects  who  did  not  use  help.  The  architect  subgroup’s  survey  responses  shared  a  number  of 
characteristics  with  the  experienced  help  and  novice  nonhelp  groups.  Based  on  survey  responses  it 
was  concluded  that  the  adaptive  on-line  help  function  was  effective.  However,  an  unexpected  finding 
was  that  about  half  of  the  participants  in  the  test  simply  did  not  use  the  help  function.  It  was 
hypothesized  that  the  nonhelp  users  carried  wiili  them  to  the  test  a  general  aversion  to  help  systems 
that  was  not  addressed  in  the  survey  because  it  was  not  anticipated. 

RECOMMENDATIONS  FOR  PRODUCT/SYSTEM: 

It  is  recommended  that  adaptive  on-line  help  should  continue  to  be  a  feature  of  USAGE  computer- 
ba.sed  instructional  systems.  In  future  revisions  of  MicroStation  Teaching  Assistant  developers  should 
consider  "repackaging”  the  help  interface  so  it  less  resembles  a  conventional  help  function,  to  which 
many  potential  users  may  have  an  aversion.  As  part  of  that  effort  the  help  screens  and  buttons  should 
be  relabeled  to  describe  more  accurately  the  kind  of  information  available  to  the  user.  Also,  the 
adaptive  mechanism  of  the  help  system  should  be  studied  further  and  developed  to  better  meet  the 
learning  needs  and  styles  of  users  who  fit  the  profile  of  the  "experienced  help  user"  and  the  "novice 
nonhelp  user." 
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ADAPTIVE  ON-UNE  HELP  FOR  EMBEDDED  INSTRUCTIONAL  SYSTEMS 


1  INTRODUCTION 


Background 

The  U.S.  Army  Corps  of  Engineers  (USACE)  has  long  used  computer-aided  design  (CAD)  for 
precision  technical  drawings.  As  the  advantages  of  using  CAD  for  concept  design  have  become  more 
apparent.  USACE  has  encouraged  its  design  architect/engineers  (A/Es)  to  use  it.  In  the  past,  traditional 
methods  of  training  A/Es  to  use  CAD  have  had  limited  success,  so  USACE  has  investigated  and  developed 
various  innovative,  nontraditional  methods.  The  results  of  this  research  indicate  that  iimovative  training 
methods  could  be  made  more  effective  and  complete  by  including  conceptual  instruction — concepts 
pertaining  to  all  CAD  systems  regardless  of  developer  or  publisher. 

lire  computer-based  (or  on-line)  tutorial  is  one  nontraditional  approach  to  CAD  education 
investigated  for  USACE  by  the  U.S.  Army  Construction  Engineering  Research  Laboratory  (USACERL). 
A  computer-based  tutorial  involves  learning  a  skill  before  using  it  in  a  job  task.  This  type  of  training  is 
usually  sequential  and  often  "intelligent,"  meaning  that  the  tutorial  program  actually  responds  to  the 
student  by  monitoring  him  or  her,  providing  feedback  for  expected  errors,  and  offering  instructional 
options  as  needed  or  desired.  One  variety  of  on-line  tutorial  is  "embedded"  instruction,  which  has  proven 
to  be  a  promising  alternative  to  traditional  methods  of  computer-based  training.  This  technology  involves 
integrating  a  tutorial  program  directly  into  the  software  environment  being  learned,  using  the  system  itself 
as  a  delivery  medium  for  the  lessons.  Users  can  learn  at  their  own  pace  on  the  same  software  they  use 
in  daily  work.  With  embedded  instruction,  the  problems  encountered  in  learning — and  the  learning 
environment  itself— are  the  same  as  those  encountered  in  actual  use. 

In  earlier  research,  USACERL  developed  a  prototype  for  embedded  instruction  in  AutoCAD,  a 
widely  used  commercial  CAD  program.'  Preliminary  tests  of  this  prototype  showed  that  embedded 
instruction  could  successfully  present  basic  techniques  and  offer  a  useful  approach  to  problem  solving. 
It  was  found  that  a  professional  audience  could  be  given  control  over  many  tutorial  options  with 
confidence  that  they  would  use  that  freedom  wisely.  An  important  finding  from  related  research  was  that 
users  who  learned  several  CAD  systems  understood  general  CAD  concepts  better  than  those  learning  their 
first  system.^  It  was  concluded  from  that  research  that  better  understanding  of  CAD  concepts  may  lead 
to  greater  CAD  productivity. 

In  addition  to  the  embedded  instruction  for  AutoCAD  mentioned  above,  USACERL  developed  a 
similar  tutorial  for  users  of  Intergraph  MicroStation,  the  Corps’  standard  CAD  software.  This  tutorial, 
called  MicroStation  Teaching  Assistant,  was  based  on  the  previous  USACERL  research,  but  included 
various  improvements.  For  example,  programmers  eliminated  the  reminder  prompt  that  told  the  user  he 
or  she  had  missed  an  opportunity  to  try  a  new  command.  It  had  been  found  that  users  sometimes 
deliberately  bypassed  such  opportunities;  many  preferred  to  experiment  freely  rather  than  foUow  exact 


'  D.S.  Shaw,  L.M.  Golish,  and  R.L.  Johnson,  InielligerU  Embedded  Instruction  for  Computer-Aided  Design  (CAD)  Systems, 
Technical  Report  (TR)  P-89A)3/ADA20181 1  (U.S.  Army  Construction  Engineering  Research  Laboratory,  Octobw  1988). 

*  D.S.  Shaw  and  L.M.  Golish.  Followup  Studies  of  Embedded  Instruction  for  CAD  Systems,  TR  P-90/1 0/ADA222509  (U.S.  Army 
Construction  Engineering  Research  Laboratory.  May  1990). 
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procedures  prescribed  by  the  tutorial.  Also,  the  error-checking  feature  was  dropped  as  an  automatic 
function  and  provided  instead  as  an  option  available  at  the  user’s  demand.  The  most  significant  change, 
however,  was  the  addition  of  adaptive,  user-sensitive  help  incorporating  information  on  general  CAD 
concepts.  When  experienced  CAD  users  invoked  the  help  function,  they  were  first  directed  to  help  screens 
dealing  with  pertinent  conceptual  issues.  On  the  oUtcr  hand,  novices  were  taken  first  to  help  screens 
telling  them  how  to  carry  out  a  specific  procedure.  Both  groups  were  free  to  view  the  other  screen  if  they 
desired.  Additionally,  the  system  was  capable  of  changing  which  kind  of  help  screen  was  offered  first, 
based  on  the  individual’s  pattern  of  help  usage. 

To  learn  whether  a  conceptual  help  function  can  improve  a  user’s  understanding  of  general  CAD 
concepts,  a  field  test  was  funded  under  the  Technology  Transfer  Test  Bed  (T^B)  program.  T^B  is  a 
USAGE  program  of  site  demonstrations  to  ensure  that  research  and  development  efforts  are  responsive 
to  customer  needs. 


Objective 

The  objective  of  this  research  was  to  investigate  the  effectiveness  of  on-line  help  in  teaching  a 
conceptual  understanding  of  CAD  systems. 


Approach 

A  general  aim  of  this  research  has  been  to  allow  students  great  freedom  to  explore  and  experiment 
in  the  context  of  an  effective  embedded  instruction  program.  While  there  are  specific  educational 
objectives  for  the  training  program,  it  was  determined  that  the  current  study  would  produce  the  most 
meaningful  results  if  the  researchers  studied  adult  students  who  set  their  own  goals.  Test  subjects  were 
studied  to  determine  which  elements  of  the  embedded  instruction  produced  significant  learning  gains. 
Although  the  researchers  are  ultimately  interested  in  the  overall  effectiveness  of  this  kind  of  training,  they 
paid  particular  attention  in  this  study  to  the  effectiveness  of  this  tutorial’s  most  important  innovation — on¬ 
line  conceptual  help. 

The  lessons  in  this  embedded  instruction  covered  the  subject  material  commonly  taught  in 
introductory  CAD  courses.  Since  the  target  constituency  for  the  end  product  is  USAGE  design 
professionals  and  technicians,  the  lessons  were  field  tested  at  three  USAGE  sites;  Engineering 
Activity/Capitol  Area,  Little  Rock  District,  and  Portland  District.  The  tests  used  a  walk-in  training  format. 
A  test  coordinator  at  each  location  administered  the  lessons  by  keeping  records  and  helping  the  subjects 
get  started.  Subjects  recorded  demographic  information  about  themselves,  including  age,  gender, 
educational  experience,  computer  experience,  and  CAD  experience.  Each  subject’s  use  of  conceptual  on¬ 
line  help  was  studied  during  the  training  sessions.  Finally,  lest  subjects  were  surveyed  for  their  opinions 
about  the  effectiveness  of  the  training  method. 

The  findings  of  this  study  were  evaluated  in  the  context  of  four  questions: 

1 .  Is  the  embedded  instructional  program  effective? 

2.  Did  the  use  of  the  help  function  differ  among  identifiable  subgroups  of  test  subjects? 

3.  Did  the  conceptual  help  screens  teach  general  CAD  concepts  effectively? 

4.  Is  the  adaptive  feature  of  the  on-line  help  function  effective? 
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Mode  of  Technology  Transfer 

In  accordance  with  the  Federal  Technology  Transfer  Act  of  1986  and  Public  Law  99-502,  the 
program  described  in  this  report  has  been  transferred  through  a  Cooperative  Research  and  Development 
Agreement  (CRDA)  between  USACERL  and  Electronic  Courseware  Systems  (ECS),  Inc.,  of  Champaign, 
IL.  ECS  refined  and  commercialized  the  USACERL  prototype  for  teaching  MicroStation,  and  it  is  now 
available  under  the  trademark  Teaching  Assistant  for  MicroStation.  The  same  concepts  developed  in  this 
research  could  be  adapted  for  a  wide  range  of  applications  by  USACE  and  private  industry. 
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2  RESF  RCH  DESIGN 


The  Instructional  Program 

Effective  computerized  instruction  is  based  on  a  knowledge  of  three  factors:  the  subject  area  itself, 
the  educational  medium,  and  the  stages  in  the  learner’s  development  of  expertise.  Since  computer-based 
instruction  attempts  to  reduce  the  need  for  human  intervention  in  training,  thorough  analysis  of  the 
learning  process  is  necessary.  The  teacher’s  usual  accommodation  of  student  needs  must  be  predicted  and 
simulated  as  closely  as  possible. 

The  subject  area  of  this  study  involved  not  only  procedural  knowledge  of  a  computer  program  but 
a  conceptual  understanding  of  how  a  computer  program  is  used  in  the  design  process.  For  designers  to 
make  the  computer  a  part  of  their  design  process,  an  expert  level  of  understanding  of  CAD  was  needed. 
Design  has  traditionally  been  thought  of  as  a  product  of  the  imagination  and  not  subject  to  computeriza¬ 
tion.  so  some  negative  reactions  to  this  approach  were  expected. 

From  initial  observations,  the  following  guidelines  were  adopted  in  planning  the  instmetion: 

1.  The  course  content  would  be  represented  belter  by  a  concept  specification  hierarchy’  than  by 
a  traditional  task  analysis  of  skills.  Different  designers  employed  different  procedural  approaches. 

2.  A/Es  would  be  encouraged  to  find  ways  to  make  the  computer  an  extension  of  their  own  design 
processes.  There  was  little  acceptance  of  the  idea  of  changing  design  practices  to  comply  with  constraints 
set  by  the  computer. 

3.  The  designer  would  be  allowed  time  and  space  for  individualized  experimentation.  Previous 
research  showed  wide  variations  in  such  needs. 

4.  In  the  interest  of  efficiency,  some  structure  would  be  followed  in  the  presentation.  The  lessons 
progressed  from  developing  general  familiarity  with  commands  to  design  application  concepts  and,  finally, 
more  detailed  procedures  of  editing  and  documentation. 

The  five  lessons  were  designed,  making  use  of  CAD  graphics  to  create  the  screen  displays. 
CONTINU,  BACK,  and  INDEX  options  appeared  on  the  screen  menu  and  were  always  active.  A  blank 
screen  was  always  available  for  practice. 

Previous  research  indicated  that  users  wanted  more  feedback  from  the  program,  and  more  control 
over  it.  In  response  to  these  requests,  the  MicroStation  version  offered  an  optional  exercise,  CHECK,  to 
monitor  the  drawing  file  and  offer  very  specific  information  to  the  u.ser.  This  program  featured  an  "expert 
critic"  that  diagnosed  user  additions  to  the  drawing  database.  If  students  satisfied  the  requirements  of  the 
task  stated  in  the  exercise,  the  user  received  an  "OK”  as  feedback.  If  the  requirements  were  not  met,  the 
expert  critic  attempted  to  determine  why  the  problem  occurred,  and  offered  corrective  feedback  if  possible. 
For  unpredicted  errors,  the  exercise  should  have  been  repeated  and  the  user  encouraged  to  try  again.  Also, 
there  was  a  reminder  to  try  the  exercise  if  no  changes  had  been  made  in  the  drawing  file.  The  expert 


M  Birenbaum  and  D  Shaw.  Ta-sk  Specification  Chart;  A  Key  to  a  Better  Understanding  of  Test  Results,"  Journal  of 
Educational  Measurrmenl,  Vol  22  (1985),  pp  219-230. 
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critic  was  available  only  if  the  user  chose  the  CHECK  function  from  the  screen  menu.  If  the  student  was 
merely  browsing  or  reviewing,  the  computer  system  would  not  interrupt. 

Extra  help  was  offered  from  some  lesson  screens  by  typing  "UC=X"  or  choosing  XHELP  from  the 
screen  menu.  This  help  was  context-sensitive;  the  same  keyboard  command  produced  different  help 
screens  from  different  lesson  pages.  At  the  beginning  of  the  embedded  MicroStation  tutorial,  the  program 
asks:  "Are  you  experienced  in  a  CAD  program  other  than  Intergraph  systems?"  The  response  to  this 
question  dictated  which  of  two  help  screens  was  offered  first  when  the  user  sought  extra  help.  Conceptual 
cxiimples  were  offered  as  primary  on-line  help  for  experienced  CAD  users,  while  the  more  traditional 
procedural  help  was  provided  to  new  CAD  users.  For  additional  help,  the  subject  could  always  obtain  the 
other  help  screen.  As  the  research  progressed,  adaptive  on-line  help  was  implemented  to  modify  the  user’s 
help  choices.  If  a  test  subject  repeatedly  demonstrated  a  preference  for  procedural  help,  regardless  of  his 
or  her  experience  level,  the  program  would  learn  to  display  that  type  of  help  screen  to  the  user  first.  An 
example  of  the  help  screen  sequence  is  shown  in  Figures  1,  2,  and  3. 

The  system  maintained  several  counters  to  keep  track  of  how  the  subject  used  the  on-line  help 
screens.  Whenever  the  subject  came  to  a  screen  with  the  on-line  help  option,  the  system  increased  a 
counter,  TP,  by  2.  For  each  actual  use  of  an  on-line  help  screen,  the  system  incremented  another  counter, 
TA,  by  1.  The  ratio  TA:TP  was  updated  whenever  the  user  returned  to  the  lesson  screen  from  the  help 
screen.  If  this  ratio  was  greater  than  0.8:1,  the  system  recorded  this  information  and  reset  the  values  of 
TP  and  TA  to  0.  If  this  ratio  exceeded  0.8:1  three  times,  the  system  reset  the  subject’s  experience  level. 
This  was  done  because  the  system  detected  that  the  subject  repeatedly  was  not  satisfied  with  the  first  on¬ 
line  help  screen  and  asked  for  a  second  one. 


Field  lest 

Three  test  sites  agreed  to  solicit  subjects  to  work  through  the  lessons  and  complete  the  survey  form 
reproduced  in  Appendix  A.  The  sites  funded  under  T^B  were  three  USAGE  offices — Engineering 
Activity/Capitol  Area,  Little  Rock  District,  and  Portland  District.  Minimal  written  documentation  was 
provided  to  accompany  the  lessons.  Each  site  named  a  coordinator  who  was  responsible  for  installing  the 
program,  helping  the  learners  start,  and  being  available  for  questions  at  appointed  hours.  The  objective 
was  to  conduct  the  testing  through  the  site  coordinators  rather  than  an  outside  project  director.  This 
arrangement  was  predicted  to  produce  more  realistic  data  even  though  it  might  also  produce  incomplete 
records  and  unexpected  differences  among  the  test  sites.  In  addition,  this  configuration  was  the  most 
economical  way  to  use  the  programs  and  was  expected  to  help  reveal  any  weaknesses  in  the  instruction. 

All  data  were  provided  anonymously.  The  site  coordinator  knew  only  the  numbers  of  test  subjects, 
but  did  not  know  the  .survey  rcsulLs.  Demographic  information  collected  included  age,  gender, 
computcr/CAD  experience,  and  educational  background.  Because  professionals  who  served  as  test  subjects 
in  related  previous  research  showed  a  clear  understanding  of  their  own  needs  and  state  of  knowledge,  it 
was  not  considered  necessary  to  give  a  final  test.  The  user’s  level  of  understanding  was  determined  on 
the  basis  of  survey  items  1.  10,  and  27  in  Appendix  A. 

Other  sets  of  survey  items  provided  information  about  learning  style,  enhancements  or  prerequisites, 
attitudes  about  computer  use  and  computer-ba.scd  instruction,  and  reactions  to  the  on-line  help.  Responses 
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LINE  PLACEMENT:  1  0  of  1  2 

CONSTRUCT  BISECTOR  LINE 

(at  the  bottom  center  of  the  paper  menu  or 

under  Const,  Bisect,  Line  on  the  sidebar  menu) 

'■  ¥ 

constructs  a  perpendicular  bisector 
to  an  existi ng  line  segment. 

A 

B 

Select  the  line  to  be  bisected  and 
move  the  cursor  until  the  bisector 
is  the  desired  length. 

Try  to  construct  a  bisector  of  li  ne  C,D. 

C 

D 

V  _ 

HELP 

y 

Figure  1.  The  Screen  With  On-line  Help  Available. 


Since  the  line  ends  are  stored  as  points  in  the  drawing  file, 
The  line  length  can  be  calculated  and  its  midpoint  located. 
The  midpoint  marks  one  end  of  the  bisector  line. 


,^,-colculoted  angle 

^^'''''^^-'Calculated  point 

. . 


stored  points'' . 

The  program  then  calculates  the  angle  to  form  the  perpendicular 
bisector,  leaving  only  the  selection  of  the  endpoint  to  the  user. 

Later  lessons  will  suggest  methods  for  insuring  point  accuracy. 


Figure  2.  A  Concept-based  Help  Screen  for  Users  With  CAD  Experience. 


First  select  the  CONSTRUCT  BISECTOR  LINE  command. 

You  may  use  the  digitizer  on  the  tablet  menu, 
select  Const  from  the  main  screen  menu.  Bisect, 

Line  from  the  second  menu,  or  type  the  command. 

Place  a  data  point  on  the  line  you  wish  to  bisect. 

Place  a  data  point  for  the  end  of  the  bisector. 


The  bisector  will  be  drawn  as  you 
move  the  cursor.  The  length  of  the 
bisector  will  equal  the  distance 
from  the  cursor  to  the  oriainal  line. 


Figure  3.  A  Procedure-oriented  Help  Screen  for  New  CAD  Users. 

to  the  survey  were  subjected  to  the  Student  Problem  Package  (SPP)  analysis'*  to  determine  whether  any 
category  of  test  subjects  answered  significantly  different  from  the  others.  The  findings  are  summarized 
in  Chapter  3.  Sorted  survey  data  arc  shown  in  Appendix  B. 


D.  Hiimisch  and  N.  Roiiiy,  SSP  Student  Problem  Package  on  the  IBM  PC  (University  of  Illinois.  1985). 
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3  FINDINGS 


Sample  Demographics 

Fifty-four  test  subjects  responded  to  the  survey.  Sixteen  of  the  subjects  were  female,  representing 
30  percent  of  the  group.  The  average  age  of  the  subjects  was  34.8.  The  division  by  discipline  was:  6 
architects,  16  civil  engineers,  8  mechanical  engineers.  4  strucmral  engineers,  11  technicians,  and  9  from 
miscellaneous  groups.  They  represented  USAGE  test  sites.  On  a  scale  of  1  (no  computer  experience)  to 
3  (more  than  1  year  of  computer  experience),  the  average  was  2.68.  Twenty-four  of  the  subjects  reported 
experience  with  AutoCAD,  a  widely  used  CAD  system  different  from  the  one  being  taught.  Four  had  a 
high  level  of  expertise  in  other  CAD  systems  and  26  had  little  or  no  experience  with  CAD  systems  other 
than  MicroStation,  the  system  used  in  this  investigation. 


SFP  Analysis 

The  SPP  analysis  (noted  in  Chapter  2)  required  that  each  item  be  scored  on  a  bipolar  scale 
(agree/disagree).  Four  options  (strongly  agree,  agree,  disagree,  strongly  disagree)  were  offered  to  the 
subjects  because  subjects  in  related  previous  research  found  it  difficult  to  make  a  choice  when  given  only 
two.  Experience  demonstrates  that  the  tendency  to  respond  to  "strongly  agree"  or  "strongly  disagree" 
depends  on  the  disposition  of  the  individual  and  cannot  necessarily  be  considered  an  absolute  indicator 
of  how  strongly  the  subject  agrees  or  disagrees.  Therefore,  for  the  purpose  of  the  SPP  analysis,  responses 
were  entered  as  1  (any  level  of  agreement)  or  0  (any  level  of  disagreement). 

The  survey  items  were  conceived  to  measure  the  user’s  attitudes  toward  computers,  computer-based 
instruction  in  general,  this  particular  computer-based  program,  the  subject  matter  being  taught  (in  this  case 
the  MicroStation  software),  and  specific  features  of  the  system.  The  survey  was  conducted  under  the 
reasonable  assumption  that  in  an  adult  professional  group  (such  as  the  ones  participating  in  the  study), 
altitude  is  an  essential  measure  of  a  training  program’s  success.  An  unfavorable  attitude  about  the  training 
program  (.or  the  system  being  taught)  is  likely  to  impede  both  the  learning  process  and  the  likelihood  of 
further  study.  The  survey  statements  were  originally  framed  as  positive  statements,  but  about  half  the 
statements  on  the  user  questionnaire  were  recast  as  negative.  A  subject’s  disagreement  with  a  negative 
statement  on  the  quesliormaire  (e.g.,  "I  had  trouble  following  the  directions...")  was  taken  to  indicate 
agreement  with  the  positive  version  (e.g.,  "I  had  no  trouble  following  the  directions..."). 

In  Table  1  and  Table  2,  and  also  for  the  purpose  of  the  analysis  sort,  negative  questionnaire  items 
were  restored  to  their  positive  phrasing,  and  the  subjects’  responses  to  those  items  were  adjusted 
accordingly  to  preserve  the  meaning  of  their  answers.  Organizing  all  the  questionnaire  items  and 
responses  in  parallel  phrasing  makes  interpretation  of  the  data  more  straightforward  to  communicate. 

The  SPP  sorts  and  arranges  the  data  from  the  most  "right"  answers  to  the  least  (top  to  bottom),  and 
from  the  highest  level  of  agreement  to  the  least  (left  to  right).  This  makes  it  possible  to  examine  the  items 
in  the  order  they  arc  prc.scnted  in  Tables  1  and  2. 

The  student  agreement  scores — the  percentage  of  the  entire  sample  agreeing  with  the  majority  of 
(he  other  participants— ranged  from  90.6  percent  to  56.3  percent,  with  a  mean  of  75.98  percent.  The 
average  raw  score  for  the  32-itcm  survey  was  24.31,  with  a  standard  deviation  of  2.93.  Examination 
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Table  1 


Items  With  75  Percent  or  Greater  Agreement 


Sorted 

Index 

Original 

Index 

Survey  Items 

Agree 

(%) 

Disagree 

(%) 

1 

11 

The  first  lesson  was  easy  enough  that  I  didn’t  feel 
frustrated. 

98 

2 

2 

13 

Trying  the  MicroStation  commands  helped  me  learn. 

98 

2 

3 

32 

I  would  like  to  see  more  advanced  lessons  made 
available. 

98 

2 

4 

2 

I  like  MicroStation. 

96 

4 

5 

8 

I  believe  that  I  could  use  MicroStation  in  my  work. 

96 

4 

6 

12 

I  like  working  with  computers. 

96 

4 

7 

17 

The  time  I  used  for  the  lessons  was  well  spent. 

96 

4 

8 

26 

I  expected  to  like  doing  the  computerized  lessons. 

96 

4 

9 

22 

I  liked  being  able  to  page  back  and  use  an  index. 

94 

6 

10 

1 

I  feel  generally  competent  with  the  MicroStation 
concepts  covered  in  the  computerized  lessons. 

93 

7 

11 

28 

I  usually  knew  when  I  did  things  wrong  in  the  les¬ 
sons. 

91 

9 

12 

30 

I  like  computer-based  instruction. 

91 

9 

13 

23 

The  computer  helps  me  consider  different  possibilities 
in  my  design. 

89 

11 

14 

24 

When  I  asked  for  help  I  wanted  to  find  procedures. 

89 

11 

15 

9 

I  had  no  trouble  following  the  directions  in  the  les¬ 
sons. 

87 

13 

16 

15 

The  lessons  were  not  too  difficult  for  me. 

85 

15 

17 

5 

The  lessons  helped  me  learn  more  quickly  than  I  do 
from  classrooms  or  textbooks. 

80 

20 

18 

16 

The  help  screens  were  very  helpful. 

80 

20 

19 

19 

A  teacher  docs  not  need  to  be  present  during  the 
computerized  lessons. 

80 

20 

20 

27 

I  understand  a  lot  about  MicroStation. 

78 

22 

21 

31 

The  help  screens  helped  me  understand  how  Micro- 
Station  works. 

78 

22 

15 


Table  2 


Items  With  Less  Than  75  Percent  Agreement 


Sorted 

Index 

Original 

Index 

Survey  Items 

Agree 

(%) 

Disagree 

(%) 

22 

29 

I  like  walk-in  training  better  than  scheduled 
classes  and  labs. 

70 

30 

23 

14 

Most  of  the  lessons  were  not  too  long  for  a 
single  session. 

70 

30 

24 

18 

I  didn’t  need  more  feedback  from  the  computer. 

65 

35 

25 

21 

By  the  end  of  the  lessons.  I  usually  only  needed 
one  help  screen  to  answer  my  questions. 

63 

37 

26 

20 

I  usually  didn’t  look  at  both  help  screens  when 
they  were  offered. 

50 

50 

- 50% - 

27 

6 

I  looked  at  the  help  screens  often. 

44 

56 

28 

10 

I  feel  capable  of  teaching  MicroStation  to  oth¬ 
ers. 

44 

56 

29 

7 

I  used  the  help  screens  more  often  at  the  begin¬ 
ning  of  the  lesson  sequence  than  I  did  at  the 
end. 

41 

59 

30 

25 

Having  previous  computer  experiences  is  not 
necessary  before  using  the  embedded  Micro- 
Station  instruction. 

41 

59 

31 

3 

Having  previous  experiences  in  drawing  and/or 
drafting  is  not  necessary  before  attempting  the 
MicroStation  instruction. 

33 

67 

32 

4 

The  lesson  documentation  is  not  necessary. 

17 

83 

For  the  different  discipline  groupings,  the  mean  raw  scores  were  as  follows: 


Architects  24.17 

Civil  Engineers  23.69 

Mechanical  Engineers  23.62 

Structural  Engineers  26.75 

Technicians  24.37 

Miscellaneous  25.0 
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of  the  different  groupings  reveals  that  females  had  a  mean  raw  score  of  24.35  and  the  males  averaged 
24.3.  Novices  averaged  24.5  and  experienced  CAD  users  averaged  24.14.  The  mean  raw  score  for  those 
under  35  years  old  was  24.24,  and  it  was  24.4  for  those  over  35.  None  of  these  differences  is  statistically 
significant. 

The  agreement  score  for  the  structural  engineers  was  statistically  higher  than  the  mean  score  for  all 
groups.  While  they  were  all  from  the  same  test  site  and  their  scores  may  reflect  a  group  dynamic  effect, 
it  is  still  interesting  to  note  that  they  differed  from  the  group  average  on  four  of  the  items  pertaining  to 
the  help  function.  They  agreed  with  items  25,  27,  and  29,  indicating  that  they  only  needed  one  help 
screen  to  answer  their  questions,  looked  at  the  help  screens  often,  and  used  the  help  screens  more  at  the 
beginning  of  the  lesson  sequence.  On  item  26  they  indicated  that  they  usually  looked  at  both  help  screens. 
All  of  these  differences  were  statistically  significant  to  at  least  a  0.95  confidence  level. 

It  is  possible  to  look  at  any  subject  record  after  the  data  is  sorted  and  deteimine  whether  the  test 
subject  has  responded  as  expected  by  the  researchers  on  any  item.  For  example,  if  student  A,  whose  score 
is  15  on  a  20-item  survey,  lists  the  following  answers: 

11111111111111100000 

he  or  she  would  be  answering  in  the  expected  pattern.  Based  on  the  sort  of  the  data  used  here  (as  men¬ 
tioned  above),  the  items  to  the  left  represent  the  ones  the  group  most  frequently  agreed  with,  and  the  five 
at  the  right  represent  the  ones  most  often  answered  "wrong"  (in  terms  of  the  researchers’  predictions)  by 
the  tested  group.  Another  subject  in  the  same  survey,  whose  answer  list  looks  like  this: 

lllOllllOllllOlOOIll 

would  be  considered  unusual.  The  score  is  still  15,  but  this  subject  has  "missed"  several  items  most  often 
agreed  with  in  this  group.  This  subject  has  also  agreed  with  the  three  items  at  the  right,  which  most  of 
the  group  did  not  agree  with.  The  SPP  would  calculate  a  high  "caution  index"  for  the  second  subject. 
Another  name  used  for  this  type  of  calculation  is  "nonconformity  index."  The  entire  sorted  set  of  data 
is  shown  in  Appendix  B.  Lines  have  been  drawn  through  flic  data  to  represent  the  expected  dividing  lines 
in  both  the  student  and  the  item  records. 

A  number  of  items  in  this  survey  proved  not  to  be  truly  bipolar  upon  analysis.  The  answer  patterns 
on  such  items  did  not  conform  well  to  the  overall  group  agreement/disagreement  pattern  for  most  items. 
The  most  obvious  cases  of  this  were  items  1,  5,  8,  14,  25,  26,  27,  30,  31,  and  32.  Items  1  and  5  had  such 
a  high  level  of  agreement,  however,  that  nonconformity  has  little  meaning.  In  the  list  of  items  cited  above 
are  four  of  the  seven  statements  about  the  help  screens — items  14, 25,  26,  and  27  in  the  sorted  list  (Tables 
1  and  2).  For  this  reason,  the  items  about  help  screens  were  analyzed  further  for  other  effects. 

The  "caution  index"  was  also  high  for  three  of  the  architects  in  the  sample.  Since  this  represents 
50  percent  of  the  architects  surveyed,  that  group  also  requires  separate  analysis.  The  other  test  subjects 
low  in  confomiity  with  the  rest  of  the  group  were  four  civil  engineers,  one  technician,  and  three  from  the 
miscellaneous  group,  none  of  which  represents  a  statistically  significant  number. 
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Item  Analysis  of  the  Architect  Group 


The  architects  displayed  an  unusual  response  pattern  over  the  32  test  items,  as  shown  in  Figure  4. 
A  comparison  of  the  architects  mean  scores  with  those  of  the  engineers  and  technicians  reveals  highly 
significant  differences  on  some  of  the  items.  Some  of  these  are  shown  in  Tables  3,  4,  5,  6,  and  7.  The 
item  numbers  refer  to  the  sorted  item  list. 

Notice  that  the  t-statistic  for  all  items  above  is  negative.  This  indicates  that  the  architects  disagreed 
with  each  statement.  They  also  differed  to  a  significance  of  97.5  percent  certainty  with  items  30  and  31, 
agreeing  that  previous  computer  or  drawing/drafting  experience  was  not  necessary  before  using  the 
embedded  MicroStation  instruction. 


Item  Analysis  by  Experience  and  Use  of  Help 

The  groups  inexperienced  in  CAD  differed  significantly  from  the  experienced  CAD  users  on  item 
14,  indicating  that  they  wanted  information  on  procedures  when  they  asked  for  help.  Experienced  users 
differed  significantly  from  the  inexperienced  on  items  20  and  28,  indicating  that  they  understood  a  lot 
about  MicroStation  and  that  they  felt  capable  of  teaching  MicroStation  to  others.  Comparison  of  all  item 
responses  by  experience  is  shown  in  Figure  5. 

Figure  6  shows  the  items  that  help  users  and  nonusers*  differed  on  significantly:  item  6  (nonhelp 
users  liked  working  with  computers  better),  item  8  (they  expected  to  like  the  computerized  lessons),  item 
10  (nonhelp  users  felt  more  competent  with  the  MicroStation  concepts  in  the  lessons),  and  item  31 
(nonhelp  users  agreed  that  previous  computer  experience  was  not  necessary).  The  nonhelp  users  agreed 
that  they  didn’t  took  at  both  help  screens  (item  26),  but  that  is  because  they  seldom  looked  at  any.  The 
help  users  agreed  significantly  that  the  help  screens  helped  them  understand  MicroStation  (item  21)  and 
that  by  the  end  of  the  lessons  they  only  needed  one  help  screen  to  answer  their  questions  (item  25). 

Some  interesting  differences  are  found  when  looking  at  the  experienced  CAD  users  and  the  novices 
by  whether  they  used  help  or  not,  and  also  when  examining  the  responses  of  help  users  and  nonusers  by 
their  level  of  experience.  There  were  12  subjects  in  each  of  two  groups:  novice  users  who  used  help  and 
experienced  users  who  used  help.  There  were  14  subjects  in  the  novice  group  that  did  not  use  help  and 
16  in  the  experienced  group  that  did  not  use  help.  Figure  7  shows  the  group  averages  over  the  32  survey 
items. 


Experienced  users  who  did  not  use  help  differed  significantly  from  experienced  users  who  used  help 
in  that  they  liked  working  with  computers,  felt  confident  with  the  concepts  covered  in  the  lessons,  liked 
computer-based  instruction,  had  no  trouble  following  the  directions  in  the  lessons,  and  learned  more 
quickly  from  the  lessons  than  from  classrooms  or  books  (items  6, 10,  12,  15,  and  17  in  Table  1).  Novices 
who  used  help  differed  from  those  who  did  not  in  that  they  gained  an  understanding  of  MicroStation 
through  help,  and  liked  walk-in  training  better  than  scheduled  classes  and  labs  (items  15,  21,  and  22  in 
Tables  1  and  2).  Novices  who  did  not  use  help  agreed  with  the  statements  that  previous  experience  in 
drawing  and  drafting  were  not  necessary  for  using  the  embedded  instruction  (item  31)  and  that  a  teacher 
did  not  need  to  be  present  in  the  classroom  (item  19)  significantly  more  often  than  novices  who  used  help. 


For  purposes  of  brevity,  test  subjects  who  did  not  use  the  help  function  are  subsequently  referred  to  as  "nonhelp  users." 
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Figure  7.  Item  Responses  by  Experience  and  the  Use  of  Help. 
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Table  3 


Analysis  of  Item  11:  'T  usually  knew  when  I  did  things  wrong..." 


Variable: 

Architects 

Engineers  and  Technicians 

Mean: 

0.67 

0.97 

Std.  Deviation: 

0.52 

0.16 

Observations: 

6 

39 

t-statistic: 

-3.03 

Hypothesis: 

Degrees  of  Freedom: 

43 

Ho:  M.1=|x2 

Significance: 

0.004 

Ha:  pl^p2 

Table  4 

Analysis  of  Item  12:  "I  like  computer-based  instruction." 


Variable: 

Architects 

Engineers  and  Technicians 

Mean: 

0.67 

0.92 

Std.  Deviation 

0.52 

0.27 

Observations 

6 

39 

t-stati.stic: 

-1.89 

Hypothesis: 

Degrees  of  Freedom: 

43 

Ho:  pl=M.2 

Significance: 

0.065 

Ha: 

Table  5 

Analysis  of  Item  15:  "I  had  no  trouble  following  the  directions..." 


Variable: 

Architects 

Engineers  and  Technicians 

Mean: 

0.67 

0.92 

Std.  Deviation; 

0.52 

0.27 

Observations: 

6 

39 

t-statistic: 

-1.89 

Hypothesis: 

Degrees  of  Freedom; 

43 

Ho:  pl=p2 

Significance: 

0.065 

Ha:  p.l^p2 
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Table  6 

Analysis  of  Item  17:  "The  lessons  helped  me  learn  more  quickly..." 


Variable: 

Architects 

Engineers  and  Technicians 

Mean: 

0.50 

0.82 

Std.  Deviation: 

0.55 

0.39 

Observations: 

6 

39 

t-statistic: 

-1.78 

Hypothesis: 

Degrees  of  Freedom: 

43 

Ho:  p.l=ti2 

Significance: 

Ha:  pl^p2 

Table  7 


Analysis  of  Item  28:  "I  feel  capable  of  teaching  MicroStation  to  others." 


Variable: 

Architects 

Engineers  and  Technicians 

Mean: 

0.17 

0.54 

Std.  Deviation: 

0.41 

0.51 

Observations: 

6 

39 

t-statistic 

-1.71 

Hypothesis: 

Degrees  of  Freedom: 

43 

Ho:  pl=p2 

Significance: 

0.094 

Ha:  p.l^p.2 

Nonhelp  users  who  were  experienced  agreed  with  the  following  statements  significantly  more 
frequently  than  nonhelp  users  who  were  novices:  item  12  (I  like  computer-based  instruction),  item  16 
(The  lessons  were  not  too  difficult  for  me),  and  item  28  (I  feel  capable  of  teaching  MicroStatii  n  to  others). 
Novice  nonhelp  users  also  differed  from  experienced  nonhelp  users  on  whether  a  teacher  need  be  present 
during  lessons  (item  19)  and  whether  previous  drawing  and  drafting  experience  was  unnecessary  (item  31). 

Help  users  who  were  novices  agreed  with  items  6. 10, 1 1, 14, 15, 17,  and  21  significantly  more  than 
help  users  who  were  experienced  in  CAD.  Novice  help  users  liked  working  with  computers,  felt  generally 
competent  with  the  MicroStation  commands  in  the  lesson,  usually  knew  when  they  did  things  wrong  in 
the  lessons,  asked  for  help  when  they  wanted  to  find  procedures,  had  no  trouble  following  the  directions 
in  the  lessons,  believed  they  learned  more  quickly  from  the  lessons  than  from  classes  or  books,  and 
indicated  that  the  help  screens  helped  them  understand  how  MicroStation  works. 

Comparing  the  experienced  group  that  used  help  with  the  novices  who  did  not  use  help,  the 
experienced  group  exhibited  significant  agreement  with  items  6, 10, 19,  30,  and  31.  This  was  interpreted 
to  mean  that  they  agreed  that  they  liked  working  with  computers,  felt  generally  competent  with  the 
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MicroStaiion  commands  in  the  lessons,  fell  lhat  a  teacher  does  not  need  to  be  present,  and  did  not  believe 
that  previous  experience  with  computers  or  drawing  and  drafting  were  necessary. 

The  experienced  group  that  did  not  use  help  agreed  with  statements  20  and  28  (indicating  they 
understood  a  lot  about  MicroStation  and  felt  capable  of  teaching  it)  significantly  more  than  the  novice 
group  lhat  did  use  help.  The  novice  group  lhat  used  help  agreed  significantly  more  than  the  experienced 
group  that  did  not  use  help  that  the  help  screens  aided  them  in  understanding  how  MicroStation  works 
(item  21). 
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4  INTERPRETATION  OF  THE  FINDINGS 


The  following  discussion  interprets  the  findings  of  the  study  in  the  context  of  the  four  research 
questions  identified  in  Chapter  1: 

1.  Is  the  embedded  instructional  program  effective? 

2.  Did  the  use  of  the  help  function  differ  among  identifiable  subgroups  of  test  subjects? 

3.  Did  the  conceptual  help  screens  teach  general  CAD  concepts  effectively? 

4.  Is  the  adaptive  feature  of  the  on-line  help  function  effective? 


Effectiveness  of  the  Instructional  System 

Based  on  the  findings  of  the  test  group  survey,  the  researchers  consider  the  instructional  program 
to  have  generally  been  successful.  On  26  of  the  32  items,  the  test  subjects  agreed  with  the  desired  or 
predicted  statement  more  than  50  percent  of  the  time.  On  21  items  there  was  more  than  75  percent 
agreement  and  on  12  items  there  was  over  90  percent  agreement  (Tables  1  and  2).  On  the  six  items 
receiving  less  than  50  percent  agreement,  four  were  not  bipolar,  indicating  that  some  factor  not  related  to 
the  success  of  the  instruction  was  a  factor.  One  of  the  two  remaining  survey  statements,  item  28  ("1  feel 
capable  of  teaching  MicroStation  to  others"),  was  clearly  an  unattainable  goal  for  the  majority  considering 
that  the  instruction  had  been  very  short  and  basic.  Item  29  ("I  used  help  screens  more  often  at  the 
beginning  of  the  lesson...")  applied  only  to  help  users  and  did  not  imply  anything  conclusive  about  the 
overall  success  of  the  embedded  instructional  program. 


Subgroups  Identified  by  Experience  and  Help  Use 

Figure  8  charts  the  agreement  patterns  of  four  subgroups.  The  top  two  quadrants  represent  help 
users  and  the  bottom  two  represent  nonhelp  users.  The  two  left  quadrants  represent  experienced  subjects 
and  the  two  right  ones  represent  novices.  Note  that  some  data  are  represented  in  the  top  and  right  margins 
of  the  figure;  these  items  are  sorted  only  on  the  basis  of  experience  (items  14,  20,  and  28  at  top)  or  help 
u.sage  (items  6,  8,  10,  21.  25,  26,  and  31),  but  not  both  criteria.  The  numbers  by  the  arrows  represent 
items  on  the  sorted  lists  (Tables  1  and  2)  arid  the  arrowheads  point  in  the  direction  of  the  group  that 
agreed  more  with  each  item. 

Experienced  users  in  general  agreed  more  than  novices  with  item  20  ("1  understand  a  lot  about 
MicroStation")  and  item  28  ("I  feel  capable  of  teaching  MicroStation  to  others").  Novices  agreed  more 
than  the  experienced  with  item  14  ("When  1  asked  for  help  1  wanted  to  find  procedures"). 

Nonhelp  users  generally  agreed  more  than  help  users  with  item  6  ("I  like  working  with 
computers..."),  item  8  ("1  expected  to  like  doing  the  computerized  lessons"),  item  10  ("I  feel  generally 
competent  with  the  MicroStation  concepts  covered  in  the  computerized  lessons"),  item  26  ("I  usually 
didn’t  look  at  both  help  screens  when  they  were  offered"!,  and  item  31  ("Having  previous  cxpcrienecs  in 
drawing  and/or  drafting  is  not  necessary  before  anempting  the  MicroStation  instruction").  Help  users 
agreed  more  than  nonhelp  users  with  item  21  ("The  help  .screens  helped  me  understand  how  MicroStation 
works")  and  item  25  ("By  the  end  of  the  lessons,  I  usually  only  needed  one  help  screen  to  answer  my 
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EH  =  experienced  help  users 
NH  =  novice  help  users 
EN  =  experienced  nonhelp  users 
NN  =  novice  nonhelp  users 
***  =  0.99  confidence  interval 
**  =  0.975  confidence  interval 
*  =  0.95  confidence  interval 
no  asterisk  =  0.90  confidence  interval 


Figure  8.  Survey  Agreement  Patterns  Sorted  by  Experience  and  Use  of  Help, 


questions").  The  nonhelp  group  would  be  expected  to  disagree  with  these  items  since  those  test  subjects 
did  not  use  help. 

The  Experienced  Help  Group 

In  Figure  8  it  is  clear  that  the  experienced  help  (EH)  users  disagreed  on  two  items  with  other 
groups;  item  6  ("1  like  working  with  computers")  and  item  10  ("I  feel  generally  competent  with  the 
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MicroStation  concepts  covered  in  the  computerized  lessons").  On  item  15  ("I  had  no  trouble  following 
the  directions  in  the  lessons")  and  item  17  ("The  lessons  helped  me  learn  more  quickly  than  I  do  from 
classrooms  or  textbooks"),  EH  users  disagreed  with  both  the  experienced  nonhelp  (EN)  users  and  the 
novice  help  (NH)  group.  Help  users  in  general  disagreed  with  nonhelp  users  on  items  6  and  10.  This 
suggests  that  help  users — particularly  experienced  ones — are  less  comfortable  with  computers  and  with 
their  ability  to  learn  from  computer-based  instruction  than  the  test  subjects  who  did  not  use  help.  Two 
of  the  four  test  subjects  who  agreed  that  they  didn’t  like  computer-based  instruction  (item  12)  came  from 
the  EH  group.  The  EH  group  disagreed  with  novice  nonhelp  (NN)  users  that  a  teacher  need  not  be  present 
(item  19),  and  with  help  users  in  general  that  previous  experiences  in  drawing  and  drafting  were  necessary 
(item  31)  before  using  the  instruction.  EH  users  also  felt  that  computer  experience  was  a  prerequisite  for 
the  instruction  (item  30).  Surprisingly,  EH  users  disagreed  with  NH  users  on  item  1 1  ("I  usually  knew 
when  1  did  things  wrong  in  the  lessons").  They  also  disagreed  with  NH  users  on  two  items  concerning 
help;  item  14  ("When  I  asked  for  help  I  wanted  to  find  procedures")  and  item  21  ("The  help  screens 
helped  me  understand  how  MicroStation  woiks"). 

The  way  the  adaptive  program  worked,  test  subjects  who  said  they  were  experienced  with  CAD  were 
directed  to  the  conceptual  help  screen  first.  Those  who  frequently  viewed  the  second  screen  would 
eventually  be  presented  with  the  procedural  screen  first.  If  they  continued  to  consult  both  help  screens, 
the  default  would  then  switch  back  to  conceptual  help.  For  those  who  did  not  want  procedural  help  it  may 
have  been  disturbing  when  the  procedural  screen  appeared  first.  Most  of  the  users  who  did  not  want 
procedural  help  were  in  the  EH  group.  Considering  that  the  EH  group  did  not  agree  that  they  understood 
a  lot  about  MicroStation  (item  20),  many  of  them  may  have  been  seeking  conceptual  help  despite  the  fact 
that  they  had  CAD  experience.  This  point  is  important  to  remember  in  the  discussion  of  teaching  concepts 
that  follow  this  section. 

The  Novice  Help  Group 

The  other  group  that  would  have  been  likely  to  experience  the  switching  of  help  screen  priority  was 
the  NH  group.  They  reported  that  they  wanted  to  find  procedural  help,  which  they  did  at  the  beginning, 
but  seemed  to  profit  from  whatever  help  was  offered.  The  NH  group  differed  from  the  NN  group  in 
desiring  a  teacher  to  be  present  (item  19)  and  in  feeling  that  previous  experience  in  drawing  and  drafting 
was  prerequisite  to  the  instruction  (item  31).  The  NH  group  agreed  more  than  the  NN  group  with  item 
15  (they  had  no  trouble  following  the  directions),  item  21  (they  felt  that  the  help  screens  helped  them 
understand),  and  item  22  (they  liked  walk-in  training  better  than  classes  and  textbooks).  The  NH  group 
appears  to  favor  computer-based  training  with  a  teacher  present  over  a  traditional  classroom  environment. 
This  group  seemed  quite  successful  with  the  instructional  program.  Their  responses  were  statistically  very 
strong,  particularly  for  items  15  and  21,  which  emphasizes  their  suitability  for  embedded  instruction  with 
adaptive  on-line  help. 

The  Novice  Nonhelp  Group 

The  NN  users  were  apparently  somewhat  uncomfortable  with  the  lessons.  Compared  with  the  NH 
and  EN  groups,  they  did  not  desire  a  teacher  (item  19)  and  did  not  consider  previous  experience  with 
drawing  and  drafting  helpful  (item  31).  They  had  some  trouble  following  the  directions  (item  15)  and 
preferred  scheduled  classes  and  labs  over  walk-in  training  (item  22).  The  NN  group  agreed  more  than 
the  EN  group  with  item  16  ("The  lessons  were  not  too  difficult  for  me").  They  did  not  like  computer- 
based  instruction  (item  12)  and  did  not  feel  capable  of  teaching  MicroStation  (item  28).  However,  they 
disagreed  more  than  the  EH  users  with  item  6  (like  to  work  with  computers),  item  10  (generally  feel 
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competent  with  the  concepts  covered  in  the  lessons),  and  item  30  (do  not  consider  computer  experience 
necessary  to  use  the  lessons).  The  learning  style  of  the  NN  group  may  be  better  suited  to  a  computer- 
based  instructional  method  other  than  this  tutorial. 

The  Experienced  Nonhelp  Group 

The  EN  group  seems  to  have  been  ideally  suited  to  this  kind  of  embedded  instruction.  To  recap 
their  responses  in  comparison  with  the  other  groups,  it  can  be  stated  that  EN  subjects  learned  quickly  from 
the  tutorial,  had  no  trouble  following  the  directions,  did  not  find  lessons  too  difficult,  liked  computer-based 
instruction,  liked  working  with  computers,  felt  competent  with  the  MicroStation  lessons,  understood  a 
lot  about  MicroStation,  and  felt  capable  of  teaching  MicroStation  to  others.  They  did  not  think  a  teacher 
needed  to  be  present,  that  previous  drawing  and  drafting  experience  was  necessary,  or  that  the  help  screens 
were  useful.  Especially  on  the  basis  of  their  responses  to  items  10,  20,  and  28,  one  can  clearly  see  that 
the  help  screens  were  not  needed  by  this  group.  EN  subjects  were  even  able  to  solve  procedural  problems 
on  the  basis  of  the  conceptual  understanding  they  already  possessed. 

Interpretation  of  Subgroup  Differences 

According  to  the  survey  responses,  the  novices  who  used  help  and  the  experts  who  did  not  use  help 
were  the  two  most  successful  groups  in  terms  of  what  they  learned  from  the  embedded  instructional 
program.  The  NH  users  were  in  78  percent  agreement  with  all  survey  items  while  the  EN  group — the 
largest  in  the  test  sample — agreed  76  percent  of  the  time.  (The  average  for  all  groups  was  also  76 
percent.)  The  NN  group  agreed  with  an  average  of  75.4  percent  of  the  survey  items,  and  the  EH  group 
agreed  with  about  74.5  percent  of  the  items.  Even  in  the  EH  group  there  was  69  percent  agreement  with 
the  set  of  items  mentioned  above  that  differed  statistically  in  disagreement  with  the  other  groups. 

Based  on  these  figures  it  appears  that  the  tutorial  instruction  was  quite  effective  for  all  groups. 
Nevertheless,  future  revisions  of  the  tutorial  should  attempt  to  make  the  instruction  more  effective  for  the 
NH  and  EH  user  groups. 

The  Architect  Subgroup 

The  architects  in  the  test  sample  exhibited  some  differences,  as  reported  in  Chapter  3.  They 
disagreed  significantly  with  the  professional  groups  on  the  following  items: 

•  "I  usually  knew  when  I  did  things  wrong  in  the  lessons"  (item  11). 

•  "1  like  computer-based  instruction"  (item  12). 

•  "I  had  no  trouble  following  the  directions  in  the  lessons"  (item  15). 

•  "The  lessons  helped  me  learn  more  quickly  than  1  do  from  classrooms  or  textbooks"  (item  17). 

•  "Having  previous  computer  experiences  is  not  necessary  before  using  the  embedded  MicroStation 
instruction"  (item  30). 

•  "Having  previous  experiences  in  drawing  aiKl/or  drafting  is  not  necessary  before  attempting  the 
MicroStation  instruction"  (item  31). 
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Based  on  these  items,  the  architects  appear  to  share  many  characteristics  of  the  EH  users.  Furthermore, 
their  disagreement  with  items  12  and  15  (from  the  above  list)  plus  item  28  ("I  feel  capable  of  teaching 
MicroStation  to  others")  was  characteristic  of  the  NN  group.  It  is  interesting  that  two  architects  came 
from  the  EH  group  and  two  came  from  the  NN  group — the  other  two  being  split  between  the  NH  and  EN 
groups.  The  architects  averaged  41.67  in  age,  which  was  significantly  older  than  the  subject  average 
(confidence  interval  0.90).  The  EH  group  averaged  38.17  years  old,  which  was  significantly  older  than 
the  EN  group  average  (32  years  old)  but  not  significantly  older  than  the  group  as  a  whole  (34.8  years  old). 
The  NH  and  NN  groups  did  not  differ  significantly  from  that  average.  Age  may  partially  explain  a 
subject’s  success  with  computer-based  instructional  programs,  but  it  cannot  be  considered  the  main  factor. 

It  has  been  suggested  that  architects  do  not  have  a  long  history  of  using  CAD  for  design  purposes, 
which  requires  a  strong  conceptual  understanding  of  computer  capabilities.  The  use  of  a  computer-based 
tutorial  may  not  seem  relevant  to  work  that  the  designer  considers  inappropriate  for  computerization. 
CAD’s  ability  to  apply  several  solutions  (e.g.,  aesthetic,  energy-related,  structural)  to  the  same  design 
problem  in  a  relatively  short  time  has  gone  unappreciated  in  the  architectural  community.  The  fact  that 
many  architects  do  use  CAD  systems  to  improve  their  designs  is  not  common  knowledge  among 
architects.  Promoting  computer-based  instruction  for  architects,  therefore,  presents  the  challenge  of 
motivating  architects  to  explore  and  use  a  technology  that  they  perceive  to  be  unrelated  to  the  way  they 
actually  woilc. 
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5  CONCLUSIONS  AND  RECOMMENDATIONS 


Conclusions 

Based  on  the  findings  of  the  user  survey,  the  embedded  tutorial  instruetion  was  effective  for  all 
groups.  However,  some  of  the  subgroups  analyzed  ap^ar  to  have  learning  styles  or  needs  that  should  be 
addressed  more  fully  in  future  revisions. 

The  novices  who  used  help  and  the  experienced  subjects  who  did  not  use  help— the  NH  and  EN  users 
respectively— were  the  two  most  successful  groups  in  terms  of  what  they  learned  from  the  embedded 
instructional  program.  Responses  of  the  novice  nonhelp  (NN)  and  experienced  help  (EH)  groups  indicate 
that  users  in  these  categories  may  not  be  completely  comfortable  with  computers  and  computer-based 
instruction  in  general,  and  this  may  interfere  with  their  receptiveness  to  it.  Also,  architects  as  a  subgroup 
appear  to  share  some  of  the  same  discomfort  with  computers  exhibited  by  the  EH  and  NN  groups. 

Teaching  CAD  Concepts 

Conclusions  about  the  adaptive  on-line  help  program  studied  here  must  include  a  note  about  how 
teaching  concepts  is  different  from  teaching  procedures.  Procedural  help  is  essentially  step-by-step 
assistance  in  the  execution  of  a  command.  A  user  is  likely  to  know  when  he  or  she  needs  procedural  help 
because  the  computer  fails  to  behave  in  the  expected  or  desired  manner.  This  idea  is  supported  by  the 
survey  results,  which  indicate  that  89  percent  of  the  test  subjects  wanted  procedural  help  when  they  asked 
for  help  (Table  1,  item  14).  On  the  other  hand  it  is  much  more  difficult  for  users  to  identify  their  specific 
need  when  they  do  not  understand  a  concept.  Consequently  it  is  difficult  to  deliver  appropriate  conceptual 
help  based  on  user  input  or  feedback. 

In  related  research  cited  previously  it  was  found  that  a  conceptual  understanding  contributed  to  more 
effective  use  of  CAD,  but  the  major  factor  contributing  to  that  understanding  seemed  to  be  previous 
experience  with  another  CAD  system.  It  seemed  that  such  experience  stimulated  an  awareness  of  global 
CAD  concepts.  The  simple  act  of  using  more  than  one  system  prompted  the  user  to  think  about 
differences  and  similarities  in  those  systems.  A  key  factor  in  this  kind  of  learning  process  seems  to  be 
the  stimulation  of  thought,  which  operates  differently  for  different  individuals.  Some  people  may  learn 
about  concepts  from  help  screens,  but  many  others  learn  them  from  examples,  pictures,  and  problems  to 
solve.  However,  it  is  known  that  the  learner  must  be  prepared  with  underlying  conceptual  knowledge 
before  new  concepts  can  be  absorbed. 

The  novices  in  this  study  who  used  help — the  NH  group — reported  that  the  help  screens  were 
effective  (Figure  8,  item  21).  Their  main  reason  for  invoking  the  help  function  was  to  learn  procedures, 
which  they  were  able  to  do  with  one  or  two  keystrokes.  Nevertheless,  the  conceptual  screens  that  the  NH 
group  viewed  may  have  built  a  foundation  for  the  conceptual  understanding  of  CAD,  and  they  were  not 
considered  to  be  a  problem.  Experienced  help  users  who  were  not  looking  primarily  for  procedural  help 
may  have  obtained  conceptual  help  screens  which,  considering  the  problems  of  delivering  conceptual  help 
noted  above,  may  not  have  been  appropriate  for  their  individual  style,  level  of  expertise,  or  specific 
conceptual  need.  To  make  it  possible  for  an  adaptive  help  function  to  "know"  a  learner’s  needs, 
researchers  must  develop  modes  of  user  input  and  feedback  that  more  accurately  reflect  the  individual’s 
needs. 
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Adaptive  Help  and  Conceptual  Understanding 


The  adaptive  feature  of  this  instructional  system  aRjears  to  account  for  the  finding  that  a  clear 
majority  of  the  test  subjects  usually  only  needed  one  help  screen  to  answer  their  questions  by  the  end  of 
the  lessons  (Table  1,  item  25).  The  system  adapted  to  their  requirements  if  they  made  use  of  the  help 
screens.  While  this  benefit  may  not  seem  overly  impressive  in  a  case  where  only  the  order  of  two  screens 
is  involved,  it  actually  can  represent  a  major  saving  of  time  and  can  improve  user  satisfaction  with  the 
system. 


Recommendations 

On  the  basis  of  this  research,  it  is  recommended  that  an  adaptive  help  function  should  continue  to 
be  a  part  of  USAGE  computer-based  instructional  systems.  While  the  findings  show  that  many  users  can 
learn  from  a  tutorial  without  consulting  help,  this  fiinction  was  considered  valuable  to  a  large  subgroup 
of  users  who  took  part  in  the  demonstration.  In  particular,  help  seems  to  make  an  important  difference 
to  novice  learners — a  key  target  group  for  any  embedded  instructional  program. 

Both  procedural  and  conceptual  help  screens  seem  to  be  effective  for  a  substantial  portion  of  the 
users  tested,  but  almost  50  percent  of  the  test  subjects  chose  not  to  use  the  help  function — an  unexpected 
result.  Experience  and  anecdotal  information  suggest  that  in  future  research  it  may  be  appropriate  to  label 
this  function  as  something  other  than  "help."  The  term  "help"  has  a  negative  connotation  to  some 
computer-experienced  individuals.  It  is  likely  that  much  of  their  experience  has  been  with  help  systems 
that  were  complicated,  counterintuitive,  or  simply  not  helpful;  many  existing  help  systems  do  not  even 
remain  in  view  when  the  user  attempts  to  follow  their  instructions.  This  may  help  explain  why  a  large 
number  of  CAD-novice  (but  computer-experienced)  users  did  not  use  the  help  in  this  study.  Assuming 
that  this  is  a  reasonable  hypothesis  to  test  in  future  research,  it  is  recommended  that  the  help  function  be 
"repackaged"  in  more  positive-sounding  terms  in  subsequent  versions  of  on-line  tutorials.  Relabeling  the 
help  screens  as,  pe  y:aps,  "picture"  or  "example"  may  help  give  the  user  a  clearer  idea  of  which  screen  he 
or  she  actually  needs. 

The  initial  help  defaults  of  this  tutorial  should  not  be  based  on  how  experienced  a  user  considers 
himself  or  herself  to  be.  A  user  with  confidence  may  perceive  himself  or  herself  to  be  experienced  after 
a  week-long  CAD  workshop,  but  another  person  with  years  of  experience  may  indicate  he  or  she  is 
inexperienced  if  he  or  she  has  not  used  the  system  for  a  few  months.  For  the  instructional  program  tested 
here,  the  authors  recommend  that  the  initial  help  default  is  for  the  procedural  screen  since  89  percent  of 
the  subjects  surveyed  wanted  procedural  help  (Table  1,  item  14).  Whenever  possible,  probability  or  other 
modeling  techniques  should  be  used  to  make  the  initial  help  settings  rather  than  user  answers  to  queries 
about  level  of  experience. 

Adaptive  management  of  the  instructional  system  should  be  studied  further.  One  of  the  initial  plans 
for  this  research  was  to  keep  a  record  of  when  the  system  adapted  (changed  the  priority  of  the  help 
screens)  for  each  user,  and  note  which  help  priority  was  set  at  the  end  of  the  tutorial.  This  was  not 
possible  with  the  version  of  MicroStation  in  which  the  tutorial  was  embedded  at  that  time,  but  an  updated 
version  of  the  program  will  support  this  type  of  user  tracking  and  recordkeeping.  Such  infonnation  will 
be  an  important  link  in  the  understanding  of  the  user’s  intent. 
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While  this  system  has  proven  to  be  generally  effective,  a  deeper  understanding  of  user  needs  and 
learning  styles  is  necessary  to  make  embedded  instructional  systems  more  flexible.  Research  beyond  the 
scope  of  this  study  would  include  both  a  deeper  coverage  of  concepts  and  access  to  a  wider  range  of 
resources.  As  the  information  needed  to  be  productive  in  automated  systems  increases,  the  capability  to 
successfully  adapt  to  the  requirements  of  an  individual  user  becomes  more  vital.  The  woric  with  adaptive 
help  capabilities  has  contributed  to  the  success  of  this  system,  and  it  represents  a  logical  point  of  departure 
for  more  advanced  research  in  tailoring  systems  to  users. 
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APPENDIX  A:  Survey  Form 


ID  NUMBER _  QUESTIUNNAIKL 

RESPOND  TO  THE  FOtJ.OWINO  STATEMENTS  BY 
CHECKING  THE  APPROPRIATE  COLUMN. 

PLEASE  DO  NOT  OMIT  ANY  ITEMS,  PICK  THE  ONE  THAT  BEST  APPLIES. 


STRONGLY 

AGREE 

AGREE 

DISAGREE 

S1RONGLY 

DISAGREE 

1. 

I  feel  generally  competent  with  the  MicroSiation 
concepts  covered  in  the  computerized  lessons. 

2. 

I  like  MicroStation. 

3. 

1  laving  previous  experiences  in  drawing  and/or 
drafting  is  necessary  before  attempting  the 
MicroStation  instruction. 

4.  • 

The  lesson  documentation  is  necessary. 

5. 

The  lessons  helped  me  learn  more  quickly  than  I  do 
from  classrooms  or  textbooks. 

6. 

I  didn't  look  at  the  help  screens  very  much. 

7. 

I  used  the  help  screens  more  often  at  the  beginning 
of  the  lesson  sequence  than  I  did  at  the  end. 

8. 

1  believe  that  1  could  use  MicroSiation  in  my  work 
projects. 

9. 

I  had  trouble  following  the  directions  in  the  lessons. 

10. 

I  feel  capable  of  leaching  MicroStation  to  others. 

11. 

The  first  lesson  was  easy  enough  that  I  didn’t  feel 
frustrated. 

12. 

1  like  working  with  computers. 

13. 

Trying  the  MicroSiation  commands  helped  me 
learn. 

14. 

Most  of  the  lessons  were  loo  long  for  a  single 
session. 

15. 

The  lessons  were  loo  difficult  for  me. 
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16.  The  help  screens  were  not  very  helpful. 

17.  The  time  I  used  for  the  lessons  was  well  spent. 


ISIRONGLY 

AGREE 


AGREE 


DISAGREE 


strongly] 
I  DiSACiREE 


18.  I  needed  more  feedback  from  the  computer  program. 

19.  A  teacher  should  be  present  during  the  computerized 
lessons. 

20.  1  looked  at  both  help  screens  whenever  they  were 
offered. 


21.  By  the  end  of  the  lessons,  I  usually  only  needed  one 
help  screen  to  answer  my  questions. 

22.  I  liked  being  able  to  page  back  and  use  an  index. 

23.  The  computer  helps  me  consider  different 
possibilities  in  my  design. 

24.  When  I  asked  for  help,  I  wanted  to  Find  procedures. 

25.  I  laving  previous  computer  experiences  is  necessary 
before  using  the  embedded  MicroStalion  instruction. 

26.  1  expected  to  like  doing  the  computerized 
instruction. 

27.  I  don’t  understand  very  much  about  MicroStalion. 

28.  I  usually  knew  when  I  did  things  wrong  in  the 
lessons. 

29.  I  like  walk-in  training  better  than  scheduled  classes 
and  labs. 

30.  I  don’t  like  computer-based  insimetion. 

3 1 .  The  help  screens  helped  me  understand  the  way 
MicroStalion  works. 

32.  I  would  like  to  see  more  advanced  lessons  made 
available. 
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APPENDIX  C:  Raw  Data  From  Survey 
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ABBREVIATIONS 


A/E 

architect/engineer 

CAD 

computer-aided  design 

CADD 

computer-aided  drafting  and  design 

CRDA 

Cooperative  Research  and  Development  Agreement 

EH 

experienced  help  (users) 

EN 

experienced  nonhelp  (users) 

NH 

novice  help  (users) 

NN 

novice  nonhelp  (users) 

SPP 

Student  Problem  Package 

T^B 

Technology  Transfer  Test  Bed 

TR 

technical  report 

USAGE 

U.S.  Army  Corps  of  Engineers 

USACERL 

U.S.  Army  Constmction  Engineering  Research  Laboratory 

i 
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USACERL  DISTRIBUTION 


Chief  of  Engineer! 

A1TN:  CBIIEC-IM-UI  (2) 

A1TN:  CEIIEC-IM-LP  (2) 

ATTN:  CBRD-L 
ATTN:  CECW 
ATTN:  CECW-O 
ATTN:  CECW-P 
ATTN;  CECW-RR 
ATFN:  CEMP 
AITN:  CEMP-M 
ATTN:  CBMP-R 
ATTN:  CEMP-C 
ATTN:  CEMP-ES 
ATTN:  CERD 
ATTN:  CERD-C 
ATTN:  CERD-M 
ATTN;  CERM 
ATTN:  DAEN-ZCE 
ATTN:  CAEN-Za 
ATTN:  DAEN-ZCM 
A'lTN:  DAEN-ZCZ 

CEIISC  22060 

ATTN:  CEIISC-ZC 
ATTN:  CENSC-FM 
A1TN:  CBHSC-PB-S 
ATTN:  CEUSC-FE-P  (2) 

US  Anny  Engineer  Dittricu 
ATTN:  Librmry  (40) 

Port  Meyer  222 1 1 

ATTN:  CENAC-EC 
Aluke  99y>6 

ATTN:  NAPEN-PL 

US  Army  Engineer  Divtstooi 
ATTN:  Libnry  (14) 

Engineer  Resource  Menegement  21005 
Topogrephic  Engineering  Center 
ATTN:  CETEC-TLSP 
High  Deeert  Area  Oftioe  93535 
Idaho  Army  Notional  Guard  83707 
ATTN:  DEH 

Mare  bland  Naval  Shipyard  94592 
ATTN:  Code  I40c 

Michigan  Depl.  of  Military  Affairs  48913 

US  Amy  Europe 

ODCS/Engineer  09403 
A1TRN;  AEAEN-FE 
ATTN:  AEAEN-ODCS 
V  Corps 

ATFN:  DEM  (II) 

VU  Corps 

ATfN:  DEH  (16) 

2 1  St  Support  Command 
ATTN:  DEM  (12) 

USA  Oertin 

ATTN;  DEH  (9) 

Allied  Command  Europe  (ACE) 

An74;  ACSGEll  0901 1 
ATIN:  SHlim/Engineer  09055 
ATEN:  AEUES  09081 
USASETAF 

ATEN:  AESE MN  D  09019 
A1TN:  ACSIiN  09168 
ATTN:  AESE-VE  09168 

m  USA.  Korea  (19) 


Ft.  Leonard  Wood.  MO  65473 

ATTN:  Canadian  Liaison  Officer 
ATTN:  German  Liaison  Staff 
ATTN:  British  Liaisoa  Officer  (2) 

A1TN:  French  Liaisoa  Officer 

USA  Japan  (USARJ) 

ATTN;  DCSEN  96343 
ATTN:  H<»JSHU  96343 
ATTN:  DEH-Okmawa  96331 

Area  Engineer.  ABDC-Area  Office 
Arnold  Air  Fdrce  Siatioo.  IN  37389 

416th  Engineer  Command  60623 
ATTN:  FaciUtica  Engineer 

US  Military  Academy  10996 
ATTN:  Fcliliea  Engineer 
ATTN:  Dept  of  Geography  A 
Environmental  Bngmg 
ATTN:  MAEN-A 

AMC  -  Dir  ,  Inst.  A  Svcs.  (23) 

DUV  ATTN:  DUV-WI  22304 

DNA  ATTN:  NADS  20305 

FOR5COM  (28) 

FORSCOM  Engr  ATTN:  Spt  Del  1S071 
ATTN:  Pacilitiea  Engineer 

HSC 

Ft  Sam  Houston  AMC  78234 
ATTN:  HSLO-F 
Fiasimoiu  AMC  80045 
ATTN:  HSHO-DEH 
Waher  Reed  AMC  20307 

ATTN:  FaciUtiei  Engineer 

IN5COM  •  Ch.  InsU.  Div. 

Arlington  Hall  Station  22212 
ATTN:  Bngr  A  Hsg  Div 
Vint  HUi  Farms  Station 
ATTN:  lAV-DEH 

USAAMCCOM  61299 
ATTN:  Library 
ATTN;  AMSMC-Rl 

Military  Disi  of  Washingotn 
ATTN:  DEH 

Cameron  Station  (3)  22314 
Fort  Lesley  J.  McNair  20319 
For  Meyer  22211 

Military  Traffic  Mgmt  Command 
Falb  Church  20315 
Oakland  Amy  Base  94^6 
Bayonne  07002 
Sunny  Poim  MOT  28461 

NARADCOM.  ATTN;  DRDNA-F  01760 

TARCOM,  Fac,  Div.  48090 

TRADOC  (19) 

HQ.  TRADOC.  ATTN:  ATEN-DEH  23651 
ATTN:  DEH 


USAIS 

Fort  Husacfauca  85613 

ATTN:  Facilities  Engineer  (3) 

Port  Ritchie  21719 

WESTCOM 

FortShafter  96858 
ATTN:  DEH 
ATTN:  A  PEN- A 

SHAPE  090S5 

ATTN:  Infrastructure  Branch,  LANDA 

HQUSEUCOM  09128 
ATTN:  ECJ4/7-LOE 

Fort  Bclvoir,  VA  22060 

ATTN:  Australian  Liaison  Officer 
ATTN:  Water  Resource  Center 
ATTN:  Engr  Studies  Coiter 
ATTN:  Topographic  Engr  Center 
ATTN:  CECC-R 

CECRUATTN:  Library  03755 

CEWES 

ATTN:  CEWES-ME  (2) 

ATTN:  CEWES-IM-DA/CPT 
ATTN:  CEWES-IM-DS 
ATTN:  Library  39180 

HQ.  XVm  Airborne  Corps  and 
Ft  Bragg  28307 

ATTN:  AFZA-DEH-EE 

Cbanute  AFB.  IL  61868 
3345  CES/DE,  Slop  27 

AMMRC  02172 

ATTN;  DRXMR-AF 
ATTN:  DRXMK-WE 

Norton  AFB,  CA  92409 
ATTN;  AFRCE-MX/DE 

Tyndall  AFB,  FL  32403 

AFESC/Engineering  &  Service  Lab 

NAVFAC 

ATTN:  Division  Offices  (11) 

ATTN:  Facilities  Engr  Cmd  (9) 

ATTN;  Naval  Public  Works  Center  (9) 
ATTN;  Naval  Civil  Engr  Ub  93043  (3) 
ATTN:  Naval  Constr  Battalion  Ctr  93043 

Engineering  Societies  Library 
New  York.  NY  10017 

National  Guard  Bureau  20310 
Installation  Division 

US  Government  FVinting  Office  20401 
Receiving/Depository  Section  (2) 

US  Army  Env.  Hygiene  Agency 
ATTN:  HSHB-ME  21010 

Americmi  Public  Works  Association  60637 

Nat’l  Institute  of  Standards  A  Tech  20899 


ROK/US  Combined  Forces  Cmd  96301 
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